{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 简易网络爬虫\n",
    "\n",
    "网络爬虫获取数据的基本原理是:\n",
    "\n",
    "1. 确定目标网站和需要爬取的数据 - 爬虫需要知道目标网站的URL,以及需要爬取网站上的哪些数据,比如文章、图片等内容。\n",
    "\n",
    "2. 发送HTTP请求 - 爬虫通过HTTP请求访问目标网页,获取网站源代码。常用的HTTP请求方法有GET和POST。\n",
    "\n",
    "3. 解析HTML - 爬虫会解析HTML源代码,通常使用解析库如Beautiful Soup,从中提取出需要的数据。\n",
    "\n",
    "4. 存储数据 - 将爬取到的数据存储到数据库或文本文件等格式中,便于后续处理和分析。\n",
    "\n",
    "5. 重复爬取 - 爬虫会递归爬取网站上的链接,遍历整个网站获取数据。需要设置一定的频率,避免过于频繁访问被网站屏蔽。\n",
    "\n",
    "6. 清洗和分析数据 - 对爬取的数据进行清洗,删除无用和重复信息。然后可以进行数据分析,得出有价值的信息。\n",
    "\n",
    "网络爬虫需要处理页面编码、会话管理、缓存、并发控制等问题,还需要遵守网站robots.txt规则,不能过度爬取避免被封IP。总体来说,爬虫通过程序化发起大规模的数据请求,自动获取网络数据,是实现数据采集的重要方式之一。\n",
    "\n",
    "## Requests 与 Beautiful Soup\n",
    "\n",
    "https://beautifulsoup.readthedocs.io/zh-cn/v4.4.0/#\n",
    "\n",
    "* Requests是Python的一个非常流行的网络爬虫工具库,主要用来发送HTTP网络请求并获取响应。Requests的主要特点包括:\n",
    "\n",
    "1. 简单易用 - Requests使用简单的Python代码就可以发送各种HTTP请求,如GET、POST等。语法简洁明了。\n",
    "\n",
    "2. 支持HTTPS - Requests可以为HTTPS请求验证SSL证书,支持所有HTTP请求方式。\n",
    "\n",
    "3. 连接池和持久连接 - Requests支持连接池,可以重用连接提高效率。并支持持久连接,实现HTTP keep-alive功能。\n",
    "\n",
    "4. 自动编码 - Requests自动处理响应内容的编码,可以很方便地获取Unicode编码后的数据。\n",
    "\n",
    "5. 自动解析JSON - Requests内置了JSON解码器,可以自动将JSON响应解析为Python字典或列表。\n",
    "\n",
    "6. 方便的请求头设置 - Requests允许添加请求头,可以定制User-Agent等信息,伪装浏览器访问。\n",
    "\n",
    "7. 建立在urllib3之上 - Requests内置了强大的urllib3,处理许多网络连接细节问题。\n",
    "\n",
    "8. 身份认证 - Requests支持HTTP基本认证、摘要认证等多种认证方式。\n",
    "\n",
    "9. 代理和流 - Requests支持设置代理,并可以获取响应内容的二进制流。\n",
    "\n",
    "综上,Requests可以大大简化Python网络爬虫的编码难度,是Python爬虫开发的重要选择。\n",
    "\n",
    "* Beautiful Soup是一个Python的网页解析库,主要用于从HTML或XML文档中提取数据。它的主要特点包括:\n",
    "\n",
    "1. 解析HTML/XML - Beautiful Soup可以自动将文档转换成Unicode编码,并解析成一个方便遍历的树形结构。\n",
    "\n",
    "2. 支持CSS选择 - 可以通过CSS选择器来提取文档中的节点,语法简洁易用。\n",
    "\n",
    "3. 支持多种解析器 - Beautiful Soup内置多种解析器,可以解析不同格式的文档。主要的解析器有Python标准库解析器、lxml HTML解析器等。\n",
    "\n",
    "4. 文档遍历 - 提供多种遍历文档树的方法,如遍历子节点、兄弟节点、父节点等,可以方便地定位提取数据。\n",
    "\n",
    "5. 搜索文档 - 支持通过标签名、属性、文本内容等来搜索匹配的文档节点。\n",
    "\n",
    "6. 修改文档树 - 可以修改文档树的节点内容、属性、标签名等,便于数据清洗。\n",
    "\n",
    "7. 输出格式化 - 可以格式化输出文档树的内容,并对输出编码设置,打印可读性更好。\n",
    "\n",
    "8. 异常处理 - Beautiful Soup会处理解析过程中的异常,保证代码的健壮性。\n",
    "\n",
    "9. 文档编码转换 - Beautiful Soup可以检测和转换文档的编码,确保解析正确。\n",
    "\n",
    "总之,Beautiful Soup是一个功能强大的网页解析库,可以大大简化网页内容提取操作,是Python网络爬虫的重要组件之一。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. 爬虫基本流程的代码实现"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import requests\n",
    "from bs4 import BeautifulSoup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "headers={\"User-Agent\": \"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:74.0) Gecko/20100101 Firefox/74.0\"}\n",
    "url=\"https://movie.douban.com/top250\"\n",
    "response=requests.get(url,headers=headers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup = BeautifulSoup(response.text)\n",
    "\n",
    "data0=soup.find_all(\"div\",class_=\"hd\")[0].find_all(\"span\")[0].text\n",
    "\n",
    "op=open(\"I:\\douban.csv\",\"w\",encoding=\"utf-8\")\n",
    "op.write(data0)\n",
    "op.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## classwork1\n",
    "爬墙商学院网站并保存数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "url_ali=\"https://abs.hznu.edu.cn/portal/xwdt/index.shtml\"\n",
    "response_ali=requests.get(url_ali)\n",
    "soup_ali = BeautifulSoup(response_ali.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "data1=soup_ali.find_all(\"div\",attrs={\"class\":\"float-left\"})[5].text\n",
    "\n",
    "op=open(\"I:\\shangxueyuan.csv\",\"w\",encoding=\"utf-8\")\n",
    "op.write(data1)\n",
    "op.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 批量获取数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "op=open(\"I:\\douban.csv\",\"w\",encoding=\"utf-8\")\n",
    "\n",
    "for i in soup.find_all(\"div\",attrs={\"class\":\"item\"}):\n",
    "    dianym=i.find(\"span\",attrs={\"class\":\"title\"}).text\n",
    "    z1=i.find(\"div\",attrs={\"class\":\"star\"}).find_all(\"span\")\n",
    "    pingfen=z1[1].text\n",
    "    renshu=z1[3].text\n",
    "    op.write(dianym+\",\"+pingfen+\",\"+renshu+\"\\n\")\n",
    "    \n",
    "op.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## classwork 2\n",
    "\n",
    "* strip():用于移除字符串头尾指定的字符（默认为空格）或字符序列。注意：该方法只能删除开头或是结尾的字符，不能删除中间部分的字符。\n",
    "\n",
    "获取商学院网站中的新闻标题与时间并保存数据\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<div class=\"float-left\" style=\"width:735px;\">学院组织学生参加2023年世界互联网大会“互联网之光”博览会 </div>,\n",
       " <div class=\"float-left\" style=\"width:735px;\">段永朝教授讲座：《数字时代的新商业思想探讨》系列第三讲——效率与公平 </div>,\n",
       " <div class=\"float-left\" style=\"width:735px;\">奋楫力发十五载，扬帆逐梦向未来 </div>,\n",
       " <div class=\"float-left\" style=\"width:735px;\">喜报|我院王淑翠教授团队的案例入选全国国际商务专业学位研究生示范案例库 </div>,\n",
       " <div class=\"float-left\" style=\"width:735px;\">学院组织学生探展2023年杭州·云栖大会 </div>,\n",
       " <div class=\"float-left\" style=\"width:735px;\">阿里巴巴商学院举行2023级国际学生迎新会 </div>,\n",
       " <div class=\"float-left\" style=\"width:735px;\">“产业互联网发展趋势、模式及案例”——中电海康窦伟博士与我院学子交流 </div>,\n",
       " <div class=\"float-left\" style=\"width:735px;\">校党委副书记杨政赴商学院走访调研 </div>,\n",
       " <div class=\"float-left\" style=\"width:735px;\">我院与杭州网营科技股份有限公司签约实践教学基地 </div>,\n",
       " <div class=\"float-left\" style=\"width:735px;\">我院召开2024年度国家项目申报动员会 </div>,\n",
       " <div class=\"float-left\" style=\"width:735px;\">张旭昆教授讲座：论民营企业长期存在的必要性 </div>,\n",
       " <div class=\"float-left\" style=\"width:735px;\">阿里巴巴商学院举行2023级本科新生欢迎会 </div>,\n",
       " <div class=\"float-left\" style=\"width:735px;\">段永朝教授讲座：《数字时代的新商业思想研讨》系列第一讲——从商业史看观念重建 </div>,\n",
       " <div class=\"float-left\" style=\"width:735px;\">南京财经大学营销与物流管理学院来我院调研交流 </div>,\n",
       " <div class=\"float-left\" style=\"width:735px;\">师徒结对薪火传 青蓝携手共成长——学院举行2023年新入职教师恳谈会 </div>]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup_ali.find_all(\"div\",attrs={\"style\":\"width:735px;\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<div class=\"float-right\">[2023-11-08]</div>,\n",
       " <div class=\"float-right\">[2023-11-07]</div>,\n",
       " <div class=\"float-right\">[2023-11-03]</div>,\n",
       " <div class=\"float-right\">[2023-11-03]</div>,\n",
       " <div class=\"float-right\">[2023-11-02]</div>,\n",
       " <div class=\"float-right\">[2023-10-30]</div>,\n",
       " <div class=\"float-right\">[2023-10-26]</div>,\n",
       " <div class=\"float-right\">[2023-10-25]</div>,\n",
       " <div class=\"float-right\">[2023-10-24]</div>,\n",
       " <div class=\"float-right\">[2023-10-20]</div>,\n",
       " <div class=\"float-right\">[2023-10-18]</div>,\n",
       " <div class=\"float-right\">[2023-10-11]</div>,\n",
       " <div class=\"float-right\">[2023-10-10]</div>,\n",
       " <div class=\"float-right\">[2023-09-20]</div>,\n",
       " <div class=\"float-right\">[2023-09-18]</div>]"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup_ali.find_all(\"div\",attrs={\"class\":\"float-right\"})[0].find_all(\"div\",attrs={\"class\":\"float-right\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "op=open(\"I:\\shangxueyuan.csv\",\"w\",encoding=\"utf-8\")\n",
    "\n",
    "biaoti=soup_ali.find_all(\"div\",attrs={\"style\":\"width:735px;\"})\n",
    "riqi=soup_ali.find_all(\"div\",attrs={\"class\":\"float-right\"})[0].find_all(\"div\",attrs={\"class\":\"float-right\"})\n",
    "for i in range(15):\n",
    "    #print(biaoti[i].text,riqi[i].text.strip(\"[]\"))\n",
    "    op.write(biaoti[i].text+\",\"+riqi[i].text.strip(\"[]\")+\"\\n\")\n",
    "op.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 跨链接获取数据\n",
    "\n",
    "* 找出链接的规律"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "url0=\"https://movie.douban.com/top250?start=\"\n",
    "url_1=\"&filter=\"\n",
    "\n",
    "soup_list_douban=[]\n",
    "for i in range(0,50,25):\n",
    "    url_z=url0+str(i)+url_1\n",
    "    response1=requests.get(url_z,headers=headers)\n",
    "    soup1 = BeautifulSoup(response1.text)\n",
    "    soup_list_douban.append(soup1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "op=open(\"I:\\douban.csv\",\"w\",encoding=\"utf-8\")\n",
    "\n",
    "for soup in soup_list_douban:\n",
    "    for i in soup.find_all(\"div\",attrs={\"class\":\"item\"}):\n",
    "        dianym=i.find(\"span\",attrs={\"class\":\"title\"}).text\n",
    "        z1=i.find(\"div\",attrs={\"class\":\"star\"}).find_all(\"span\")\n",
    "        pingfen=z1[1].text\n",
    "        renshu=z1[3].text\n",
    "        op.write(dianym+\",\"+pingfen+\",\"+renshu+\"\\n\")\n",
    "\n",
    "op.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## classwork 3\n",
    "\n",
    "获取多页商学院的新闻标题与时间"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://abs.hznu.edu.cn/portal/xwdt/index_2.shtml\n",
      "https://abs.hznu.edu.cn/portal/xwdt/index_3.shtml\n",
      "https://abs.hznu.edu.cn/portal/xwdt/index_4.shtml\n"
     ]
    }
   ],
   "source": [
    "url_ali0=\"https://abs.hznu.edu.cn/portal/xwdt/index\"\n",
    "url_ali1=\".shtml\"\n",
    "response_ali=requests.get(url_ali)\n",
    "soup_ali = BeautifulSoup(response_ali.text)\n",
    "soup_list_ali=[soup_ali]\n",
    "\n",
    "for i in range(2,5):\n",
    "    print(url_ali0+\"_\"+str(i)+url_ali1)\n",
    "    response_ali=requests.get(url_ali0+\"_\"+str(i)+url_ali1)\n",
    "    soup_ali = BeautifulSoup(response_ali.text)\n",
    "    soup_list_ali.append(soup_ali)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "op=open(\"I:\\shangxueyuan.csv\",\"w\",encoding=\"utf-8\")\n",
    "\n",
    "for i in soup_list_ali:\n",
    "    biaoti=i.find_all(\"div\",attrs={\"style\":\"width:735px;\"})\n",
    "    riqi=i.find_all(\"div\",attrs={\"class\":\"float-right\"})[0].find_all(\"div\",attrs={\"class\":\"float-right\"})\n",
    "    \n",
    "    for j in range(len(biaoti)):\n",
    "        op.write(biaoti[j].text+\",\"+riqi[j].text.strip(\"[]\")+\"\\n\")\n",
    "op.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
